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(54) Object detecting apparatus In which the position of a planar object is estimated by using 
hough transform 



(57) Based on left and right images that have been 
taken by image pickup devices, a correlation processing 
section obtains parallax data as three-dimensional data 
of objects. At processing start time, a plane estimating 
section estimates the position of a planar object such as 
a road or a floor by utilizing the Hough transform based 
on the parallax data that have been obtained success- 
fully for part of rectangular segments of the images, and 
interpolates the parallax data by using the position of 
the planar object. At a time point after the processing 
start time, a parallax variation detecting section detects 
an object by comparing current parallax data with the 
parallax data at the processing start time. 
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t start is called a° inrtiai parallax image, to distinguish it from a parallax image at time t (> t S TART)- 

By comparing parallaxes of the same block 302 of a parallax image at time t (present time) and that at time t START 
(past time), a parallax variation detecting section 204 judges that a new object has appeared or an object has disap- 
peared in a block 302 with which a parallax variation larger than a given threshold has occurred. 
5 Now, the Hough transform will be described with reference to Fig. 5. A straight line 501 on the x-y plane is 

expressed as 

p 0 =xcos6 0 +ysin6 0 (4) 

w where po is the length of a perpendicular 502 dropped from the origin O toward the straight line 501 and 6 0 is an angle 
formed by the perpendicular 502 and the x axis. 

Equation (5) is defined for each of a series of points (x jt yj), i = 0, 1 , 2, ... on the x-y plane: 

p = Xjcose + yjSine (5) 

15 

On the p-6 plane, Equation (6) is expressed as a composite sinusoidal function called a Hough curve as shown in Fig. 
6, and represents every straight line passing through the point (xj, yj. Where the points (x^, yj) are approximately located 
on a straight line, their Hough curves on the p-6 plane cross each other in the vicinity of a single point 601 as shown in 
Fig. 6. The straight line on the x-y plane (see Equation (4)) is determined by point 601 (p 0 , 9 0 ) where most Hough 

20 curves cross each other. The above operation is called straight line fitting by the Hough transform. Since the straight 
line fitting by the Hough transform is a straight line detecting method with recognition of a general tendency of a series 
of data points, a straight line can be detected even if the data points include points that are not located on that line. 

The above object detecting apparatus is required to be able to correctly estimate the position, in a three-dimen- 
sional space, of a relatively large planar object such as a road, a floor, or a wall which is to serve as a reference for 

25 object detection judgment. 

In the above object detecting apparatus, when rectangular segments of left and right images 301 and 401 are cor- 
related with each other by the correlation processing section 203, a segment of the right image 401 which makes the 
similarity evaluation value C smallest is employed as a corresponding segment. However, there may occur a case that 
because of a small number of features, a segment that makes the similarity evaluation value C smallest cannot be 

30 obtained clearly for part of blocks 302 of a planar object and hence parallax data cannot be calculated therefor. This 
means that the object detecting apparatus cannot perform correct detection which apparatus operates on the assump- 
tion that parallax data are obtained almost completely in initial and current parallax images. 

Further, in the convention correlation processing, a parallax-related distance measurement range 703 (see Fig. 7) 
is set in consideration of an object closest to and an object most distant from image pickup devices 702, and is fixed 

35 irrespective of the position in an image. However, in almost all cases, an object to be detected is located closer to the 
image pickup devices 702 than a plane object 701 such as a road, a floor, or a wall is, and the plane object 701 does 
not extend perpendicularly to the optical axes of the image pickup devices 701 . This means that the amount of calcula- 
tion is unduly large in the conventional technique because the parallax determination is performed in vain in a range 
which includes no object for distance measurement 

40 

SUMMARY OF THE INVENTION 

A first object of the present invention is to correctly estimate a three-dimensional position of a planar object by a 
global straight line approximation technique utilizing the Hough transform, without being affected by objects other than 
45 the planar object. 

A second object of the invention is to perform object detection with high accuracy by estimating the position of a 
planar object and inserting proper data for undetermined parallax data of the same plane object by interpolation. 

A third object of the invention is to reduce the amount of correlation calculation by optimizing the parallax determi- 
nation range for each block position of an image based on information of an estimated position of a plane object. 
so To attain the above objects, the invention has a general feature that the position of a planar object is estimated by 
Hough transform processing based on successfully measured parallel data. 

According to a first aspect of the invention, there is provided an object detecting apparatus comprising: 

a plurality of image pickup means disposed at a predetermined interval, for producing a plurality of images: 
55 a correlation processing section for producing three-dimensional data of objects according to triangulation through 
correlation between the plurality of images; and 

a plane estimating section for estimating a position of a planar object in a three-dimensional space by performing 
Hough transform on the three-dimensional data. 
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mined ones of the same planar object through an interpolating operation by using the estimated position of the planar 
object. As a result, an object that is distant from the planar object can be detected more accurately. 

According to still another aspect of the invention, there is provided an object detecting apparatus comprising: 

a plurality of image pickup means disposed at a predetermined interval, for producing a plurality of images; 
means for estimating a position of a planar object in a three-dimensional space based on the plurality of images; 
and 

means for producing three-dimensional data of an object except for an object that is more distant from the plurality 
of image pickup means than the planar object is. 

With this configuration, the amount of correlation calculation can be reduced because three-dimensional data is not 
measured for an object that is more distant from the image pickup devices than a planar object such as a road, a floor, 
Of a wall is. 

>5 BRIEF DESCRIPTION OF THE DRAWINGS 

Ftg i is Q flow diagram showing the configuration of an object detecting apparatus having a plane estimating sec* 
bon ©ccorckng to a first or third embodiment of the present invention; 

Ftg 2 is a flow diagram showing the configuration of a conventional object detecting apparatus; 
a* Fig 3 iftuctrates how blocks are arranged in a left image, i.e.. reference image; 

Ftg 4 illustrates a parallax search area of a right image in the conventional technique; 
Fig 5 shoa/s straight line parameters used for the Hough transform; 
Fig 6 shews an example of Hough curves; 

Fig 7 illustrates a general positional relationship between image pickup devices and a planar object; 
25 Fig 8 is a flowchart showing a plane estimating scheme according to a second embodiment of the invention; 

Ftg 9 shews a three-dimensional reference coordinate system used in the second embodiment; 

Ftg 10 shows fitting of a plane passage straight line in the second embodiment; 

Ftg 1 1 illustrates straight line approximation with the Hough transform in the second embodiment; 

Ftg 1 2 shows a result of step 801 of the plane estimating scheme according to the second embodiment; 
so Fig. 13 shows a result of step 802 of the plane estimating scheme according to the second embodiment; 

Fig 14 shows a distance correction line in the second embodiment; 

Fig 15 shows fitting of the distance correction line in the second embodiment of the invention; and 

Fig. 16 shows a result of step 803 of the plane estimating scheme according to the second embodiment. 

35 DESCRIPTION Q F THE PREFERRED EMBODIMENT? 

Embodiments of the present invention will be hereinafter described with reference to the accompanying drawings. 

Embodiment 1 

40 

Fig. 1 shows the configuration of an object detecting apparatus according this embodiment. In this object detecting 
apparatus, of left and right images 106 and 107 taken by image pickup devices 101 and 102. the left image 106 is 
divided into M (horizontal) x N (vertical) blocks. A correlation processing section 103 acquires a parallax S(X, Y) as 
three-dimensional data of an object on a block-by-block basis. A plane estimating section 104 estimates the position of 

45 a planar object by using parallaxes successfully measured by the plane estimating section 104. A parallax variation 
detecting section 105 extracts a block with which the parallax of a parallax image 108 at time t has varied from that of 
an initial parallax image 109 at processing start time t START (< t) by more than a threshold TH, and judges that there 
exists a new object at a position indicated by the block. 

The operation of the object detecting apparatus having the above configuration will be described with reference to 

so Fig. 1 . Separate descriptions will be made of a stage where images are input at processing start time t START and a stage 
where images are input at time t (> t S TAFrr)- 

At the first stage, images taken at time t START are input to the correlation processing section 1 03, where the left and 
right images 106 and 107 are correlated with each other and information of positional deviations between the images 
106 and 107. that is. parallaxes S are produced. The parallax is a kind of three-dimensional data. The parallaxes S(X. 

55 Y) of respective blocks obtained by dividing the left image 106 are input to the plane estimating section 104, which esti- 
mates the position of a plane object in the real space by using successfully measured parallaxes S(X, Y), and outputs 
an initial parallax image 109 including the parallaxes S(X, Y) of the respective blocks. Further, the plane estimating sec- 
tion 104 performs interpolation to produce parallaxes for blocks with which the parallax measurement failed in the above 
attempt. 
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p E (Y), they are written as 

e K ( v) = e« + e F ( V) (8) 

p*m = P*( n + Pf m 

where 6 R and pr(Y) are their highly reliable values. That is, at the present stage the plane passage straight lines not 
necessarily exist on the same plane, and this will be corrected by the following operations. 

w [Step 802: Correction of slopes of plane passage straight lines] 

If all the plane passage straight line existed on the same plane, the angles 0k(Y) formed by the perpendiculars to 
the Y axis (projections onto the S-X plane) of all the straight lines and the X axis should have the same value. Therefore, 
a center value 6 R is determined by sorting magnitudes of the angles 6 K (Y), anc * all the angles 6 K (Y) are replaced by the 
is center value 6 R . Thus, the angles 0|<(Y) formed by the perpendiculars of the respective plane passage straight lines and 
the X axis are now represented by the most reliable slope angle e R . Stated in terms of formulae, Equation (9) corre- 
sponds to the case where the angles 6|<(Y) of the respective plane passage straight lines include the error components 
0e(Y), and Equation (10) corresponds to the case where the error components 6^(Y) are removed from the angles 
6 K (Y) of all the straight lines by replacing the angles Qk(Y) with the most reliable angle 6 R . 

20 

^' rj - sin (6^0^ V)) smfe^e^YO) w 

25 

cos 9 R p o+p AY) 
S(X ,Y) = - . ,f (10) 

v 1 sin e R sin B R v ' 

As a result of the above operation, the slopes of the plane passage straight lines, which were different from the 
slope of the plane as defined by the angle of a perpendicular to the Y axis, are now corrected. At the present stage, all 
the plane passage straight lines have the same slope as shown in Fig. 13 in which they are drawn in the S-X-Y refer- 
ence coordinate space. 

[Step 803: Distance correction of plane passage straight lines] 



30 
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Where there exist two non-parallel planes in a three-dimensional space, they should have a straight crossing line. 
Based on this fact, as shown in Fig. 14, a straight line formed by a plane 1401 to be estimated and a Y-p K plane 1402 
40 is determined by utilizing the Hough transform. The p K axis is an axis that is inclined from the X-axis by 6 R . Distances 
p K (Y) from the Y axis of the respective straight lines are plotted on the Y-p K plane as shown in Fig. 15. Plotted points 
(Y, p K ) on the Y-p K plane are subjected to the Hough transform according to Equation (11). 
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SO 



p' - Ycos8* + p K (Y)sin6\ 0' starl ^ 0' £ 0 e nd 01) 

After a point (p' s , 0's) where most Hough curves cross each other on the p'-0' plane is found, a straight line 1 501 is 
determined by using Equation (11). Since the straight line 1501 indicates distances from the Y axis which are consid- 
ered most suitable for the respective plane passage straight lines, it is called a distance correction line. In Fig. 14, the 
distance correction line is denoted by numeral 1403. The distance correction line is expressed as 



K * v ' sine s sine s 



By correcting the distances pkPO from the Y axis of the respective plane passage straight lines into p R (Y) by using 
the distance correction line 1403. Equation (10) is changed to 
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receiving a plurality of images that are produced by a plurality of image pickup means disposed at a predeter- 
mined interval; 

calculating three-dimensional data S(X. Y); 1 £ X £ M, 1 £ Y £ N for each of rectangular segments obtained by 
dividing the received images into M parts in a horizontal direction and into N parts in a vertical direction through 
correlation between the received images; 

determining a plane passage straight line L(Y) by performing Hough transform on M three-dimensional data 
corresponding to rectangular segments belonging to a horizontal area GL{Y) of the received images; 
performing the determining step on all horizontal areas GL(Y), 1 £ Y £ N, to thereby determine N plane pas- 
sage straight lines L(Y), 1 s Y £ N; and 

estimating a position of a planar object in a three-dimensional space based on the N plane passage straight 
lines L(Y), 1 £ Y £ N. 

The plane estimating method according to daim 2, further comprising the steps of: 

calculating an angle e(Y) of a perpendicular to both of a Y axis and each of the N plane passage straight lines 

L(Y), 1 <£ Y £ N, to obtain N angles 6(Y), 1 £ Y £ N; 

calculating a central angle value 0r of the N angles 6(Y), 1 £ Y £ N; and 

replacing all of the angles 6(Y), 1 £ Y £ N with the central angle value e R , to obtain N angle-corrected plane 
passage straight lines L*(Y), 1 £ Y £ N, 

wherein the estimating step estimates the position of the planar object based on the N angle-corrected plane 
passage straight lines L'(Y), 1 £ Y £ N. 

The plane estimating method according to claim 3, further comprising the steps of: 

calculating a distance p(Y) between the origin of a coordinate system of the three-dimensional data S(X. Y) 
and each of the N angle-corrected plane passage straight lines L'(Y). 1 <; Y <> N, to obtain N distances p(Y), 1 
£ Y £ N; 

determining a distance correction line by performing Hough transform on the N distances p(Y), 1 <> Y £ N; and 
translating the N angle-corrected plane passage straight lines L'(Y), 1 £ Y £ N so that they intersect the dis- 
tance correction line, to obtain N distance-corrected plane passage straight lines L°(Y), 1 £ Y <> N, 

wherein the estimating step estimates the position of the planar object based on the N distance-corrected 
plane passage straight lines L"(Y), 1 ^ y <, N. 

An object detecting apparatus comprising: 

a plurality of image pickup means disposed at a predetermined interval, for producing a plurality of images; 
means for producing three-dimensional data of objects for respective rectangular segments of the plurality of 
images; 

a plane estimating section for estimating a position of a planar object in a three-dimensional space based on 
the three-dimensional data; 

means for producing, by using the position of the planar object, three-dimensional data for part of the rectan- 
gular segments for which three-dimensional data could not be produced, to obtain interpolated three-dimen- 
sional data; and 

means for detecting an object that is located at a position different than the planar object when viewed from the 
plurality of image pickup means based on the interpolated three-dimensional data. 

An object detecting apparatus comprising: 

a plurality of image pickup means disposed at a predetermined interval, for producing a plurality of images; 
means for estimating a position of a planar object in a three-dimensional space based on the plurality of 
images; and 

means for producing three-dimensional data of an object except for an object that is more distant from the plu- 
rality of image pickup means than the planar object is. 
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FIG. 5 
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FIG. 9 
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(54) Object detecting apparatus in which the position of a planar object is estimated by using 
hough transform 



(57) Based on left and right images that have been 
taken by image pickup devices, a correlation processing 
section obtains parallax data as three-dimensional data 
of objects. At processing start time, a plane estimating 
section estimates the position of a planar object such as 
a road or a floor by utilizing the Hough transform based 
on the parallax data that have been obtained success- 
fully for part of rectangular segments of the images, and 
interpolates the parallax data by using the position of 
the planar object. At a time point after the processing 
start time, a parallax variation detecting section detects 
an object by comparing current parallax data with the 
parallax data at the processing start time. 
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